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1. Introduction 

Achieving high response rates in sample surveys has become increasingly more difficult in recent 
years. Atrostic, Bates, Burt, and Silberstein (2001) show that at the end of the twentieth century, the rates 
of nonresponse were increasing for in-person household surveys conducted by the U.S. Census Bureau. 
The issue is of even greater concern in random digit dial (RDD) telephone surveys. Curtin, Presser, and 
Singer (2000) and Steeh et al. (2001) examine changes in response rates in RDD surveys, but the Survey 
of Consumer Attitudes (SC A) is the only national RDD survey they review. No response rates for RDD 
surveys conducted after 1999 are given in these articles. However, steep declines in RDD survey response 
rates after 1999 have been reported by Curtin, Presser, and Singer (2005). 

Groves and Couper (1998) describe a host of factors that may affect response rates in household 
surveys. For RDD surveys, the most important of these factors are those related to the study 
characteristics (e.g., burden, salience, sponsorship, and content), the general survey environment (e.g., 
call screening technology, level of telemarketing and other types of calls to the household, and the 
economy), and the survey methodology (e.g., use of advance letters, refusal conversion attempts, and 
monetary incentives). Curtin, Presser, and Singer (2000) present evidence that the level of effort used to 
gain survey participation in the SCA increased over approximately 18 years. More specifically, they show 
that the mean number of call attempts to complete an interview more than doubled and that there was an 
increase in refusal conversions, resulting in an increase of about 25 percent in interview time to complete 
a survey. 1 By 1996, monetary incentives were introduced experimentally in the SCA and by 1999, 
incentives were used for all possible respondents in an effort to keep response rates from falling to 
unacceptable levels. 



The National Household Education Surveys Program 

The National Household Education Surveys Program (NHES) is an RDD survey developed by the 
National Center for Education Statistics (NCES) in the Institute of Education Sciences, U.S. Department 
of Education. It is designed to collect information on important educational issues through telephone 
surveys of households in the United States. NHES has enabled NCES to gather data on a wide range of 
issues, such as early childhood care and education, children’s readiness for school, parent perceptions of 
school safety and discipline, before- and after-school activities of school-age children, participation in 
adult and continuing education, parent involvement in education, school choice, homeschooling, and civic 
involvement. NHES uses computer-assisted telephone interviews (CATI) and has been conducted by 
Westat in 1991, 1993, 1995, 1996, 1999, 2001, and 2003. 

NHES provides data on the populations of special interest to NCES and education researchers as 
defined by age and/or grade in school for each child survey and for persons ages 16 and older who are not 
enrolled in grade 12 or below for each adult survey. It targets these populations using specific screening 
and sampling procedures. Specific age or grade ranges for a given survey are determined by the survey 
topic and the research questions formulated for the specific survey administration. 

NHES provides national cross-sectional estimates for the 50 states and the District of Columbia. 
The NHES design also yields estimates for subgroups of interest for each survey, as defined by age or 
grade for children, educational participation status for adults, and Black and Hispanic origin for all 
populations of interest. In addition to providing cross-sectional estimates, NHES is also designed to 
provide estimates of change over time in key statistics. The survey instruments are designed to address 



1 Curtin, Presser, and Singer (2000) cite an increase from 2.1 hours in 1981 to 2.7 in 1996; these 2 years were used for this comparison because 
the average interview lengths for these two administrations were comparable. 
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the selected issues in sufficient detail so that analyses can be performed to help explain the phenomena of 
interest. 

This report describes an experiment conducted in NHES:2003 that was designed to examine the 
effect of different mailings and monetary incentives. The goal of the experiment was to determine if 
monetary incentives should be used in future surveys, and to identify the effectiveness of various levels of 
incentives in gaining increased initial cooperation, refusal conversion, and overall unit response rates. The 
2003 administration was conducted from January 2 through April 13, 2003. Data were collected using a two 
stage sampling procedure, where households were selected and then screened for eligible subjects for the 
topical surveys. Then, within each household, eligible subjects were sampled and topical interviews were 
conducted about them. In the NHES:2003 Screener, household members were enumerated and 
demographic and educational information that determined eligibility for the two topical surveys was 
collected. The NHES:2003 topical surveys were the Parent and Family Involvement in Education Survey 
(PFI) and the Adult Education for Work-Related Reasons Survey (AEWR). The populations of interest in 
the surveys were children enrolled in kindergarten through 12th grade for PFI and persons age 16 or older 
who were not enrolled in grade 12 or below, not institutionalized, and not on active duty in the U.S. aimed 
forces for the adult education survey. The Screener was brief and took an average of 3.5 minutes to 
complete, while the PFI interview took an average of 24. 1 minutes to complete and the adult education 
interview took an average of 16.8 minutes to complete (data not shown). 

The monetary incentive experiment (detailed later) focused primarily on the initial contact, or 
Screener, because gaining participation at this initial stage is the most difficult component of RDD 
surveys (Groves and Couper 1996; Maynard and Schaeffer 1997). Table 1 shows the number of 
completed screener interviews and the response rates for NHES data collections from 1991 to 2003. A 
graphical depiction of the NHES Screener response rates between 1991 and 2003 is given in Figure 1. 

The response rates are weighted by the base weights of the sampled telephone numbers to account for 
differential selection probabilities used in the surveys. 

The Screener unit response rates in the first two administrations (1991 and 1993) were more than 
80 percent, but then fell in 1995 and 1996. Brick and Collins (1997a) show that much of this decrease is 
due to changes that increased the length and content of the screening interview. Brick and Collins (1997b) 
also report on an experiment that confirmed these findings. 2 In 1999, the length and content of the 
Screener were revised to be more consistent with the earlier surveys and the response rates rose to 74 
percent. The Screener response rates for 2001 and 2003 declined despite the fact that the length and 
content of the Screener were very similar to 1999. 

This decrease in Screener response rates occurred despite increases in the levels of effort used in 
NHES over the years. These increases were in the number of call attempts, the number of refusal 
conversions, and the use of mailings. The levels of effort for the earlier surveys (1991, 1993, 1995, and 
1996) are summarized in Brick and Collins (1997a), Nolin et al. (2000) for the 1999 survey, Hagedorn et 
al. (2003) for the 2001 survey, and Hagedorn et al. (forthcoming) for the 2003 survey. These reports 
discuss methods to encourage participation in NHES such as: use of letters, messages left on answering 
machines, refusal conversion attempts, additional call attempts for different types of outcomes, special 
interviewer training, and monetary incentives. 

In NHES: 2001, addresses that could be used for mailing purposes (referred to as mailable 
addresses) were obtained for the sampled telephone numbers from either of two commercial firms 
(referred to as a mailable address). Addresses were obtained for about 48 percent of sampled telephone 
numbers (about 85 percent of known residential telephone numbers in the sample). A first class letter 



The Brick and Collins (1997b) study included a comparison of response rates for a brief Screener containing a “screen-out” question with an 
expanded Screener that included full enumeration. 
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explaining the puipose of the survey and encouraging participation in the study was mailed to a random 
sample of one-half of the households with mailable addresses. The weighted unit response rate for 
numbers with addresses to which letters were mailed was 75 percent and for those numbers with addresses 
to which letters were not sent the rate was 70 percent. (By comparison, the unit response rate for numbers 
without addresses was 55 percent.) 

Table 1. Weighted unit response rates and percentage distribution of reasons for unit 
nonresponse for the NHES Screener: Selected years, 1991-2003 



Year of survey 


Number of 
completed 
screeners 


Overall unit 
response rate 
(percent) 


Reasons for unit nonresponse (percentage distribution) 


Refusals 


Maximum 

calls 1 


Other 

nonresponse 2 


1991 


60,322 


81.0 


84 


7 


9 


1993 


63,844 


82.1 


68 


15 


18 3 


1995 


45,465 


73.3 


84 


9 


7 


1996 


55,838 


69.9 


83 


10 


7 


1999 


55,929 


74.1 


76 


17 


7 


2001 


48,385 


67.5 


74 


18 


8 


2003 


32,049 


61.7 


76 


16 


8 



1 “Maximum call” cases are those that received at least eight call attempts during which contact was made with a person yet the 
Screener was not completed. 

2 “Other nonresponse” includes language problems, a portion of residential no-answer and answering machine calls, and other 
forms of nonresponse. 

3 The NHES: 1993 percent of “other nonresponse” cases is higher than that in other surveys. The lower rate of refusals in 
NHES: 1993 and the generally higher response rate are indicative of the fact that less refielding of other nonresponse cases was 
needed prior to ending data collection with an acceptable Screener response rate. 

NOTE: To avoid any differences in rates that might be attributable to the calculation method, all unit response rates given here 
were calculated using the business office method. The official rates for 2001 and 2003 use the survival method (see appendix A 
for details on the methods for computing response rates). The screening procedure (i.e., number of household members 
enumerated) in each data collection differed according to the sample requirements of the topical surveys conducted in the specific 
year. Percents may not sum to 100 because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 
Program, Selected years, 1991-2003. 
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Figure 1. Weighted unit response rates for the NHES Screener: Selected years, 1991-2003 



Unit response 
rate (%) 




1991 1993 1995 1996 1999 2001 2003 

Year of survey 



NOTE: To avoid any differences in rates that might be attributable to the calculation method, all unit response rates given here 
were calculated using the business office method. The official rates for 2001 and 2003 use the survival method (see appendix A 
for details on the methods for computing response rates). The screening procedure (i.e., number of household members 
enumerated) in each data collection differed according to the sample requirements of the topical surveys conducted in the specific 
year. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 
Program, Selected years, 1991-2003. 
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The results of the advance mailing in NHES:2001 provided further evidence that sending an 
advance letter increased Screener response rates in the survey. Previously, Brick and Collins (1997b) 
reported this result from an experiment conducted in a pretest for NHES: 1996. This finding is not 
consistent with experiments in some other RDD surveys (e.g.. Singer, Van Hoewyk, and Maher, 2000) 
where an advance letter did not increase response rates. The NHES advance letter is sent in an official 
U.S. Department of Education envelope, which may increase the likelihood that respondents open the 
mail and give greater legitimacy to the survey, as noted in Dillman (2000). This issue will be addressed 
later in the analysis of the incentive experiment findings. 

A primary focus of the current experiment, the effectiveness of mailings to households that 
initially refuse to participate, was also examined in the 2001 NHES survey. For the 2001 survey, refusal 
conversion letters (i.e., letters intended to persuade respondents to change their minds and complete the 
survey) were sent via FedEx or Priority Mail to 40 percent of the eligible telephone numbers for which 
addresses were available. Of those that received refusal conversion letters, 55 percent completed the 
Screener, compared to 38 percent of those that were not sent a letter. This difference must be interpreted 
with caution, however, as the receipt of a refusal conversion letter is confounded with the availability of 
address information. In the 2003 NHES experiment discussed in this report, a more systematic approach 
was taken to examining the effectiveness of sending a letter in increasing the probability of refusal 
conversion. 

In the next section, some of the literature on the use of incentives in surveys is reviewed, 
concentrating on RDD surveys to the extent possible; the design of the experiment in NHES:2003 is then 
described. Next the findings of the experiment are presented, including information on the effects of 
varying incentive and mailing conditions on response rates, costs, and quality. The conclusion provides 
some discussion of the implications of these findings for future RDD surveys, and NHES in particular. 



2. Incentives in Surveys 

Incentives have been used as a way of improving response rates in surveys for decades; yet the 
theory supporting the use of incentives is still not fully established. One theory proposed to explain the 
effectiveness of incentives is social exchange (Dillman, 2000), which suggests that giving an incentive to 
a household is considered by the household as an act that must be reciprocated. Responding to the survey 
is the reciprocal act. Monetary exchange theory is also sometimes invoked to explain responses when 
incentives are used. This theory suggests the household may be more likely to respond to the survey 
because the incentive is payment for this response. Groves, Singer, and Corning (2000) suggested a more 
complex model in which incentives may be used to compensate for other survey characteristics that may 
diminish likelihood of responding, such as long length (i.e., high respondent burden), content that lacks 
relevance to a respondent, or sponsorship by an organization that may be unknown to a respondent. 

Singer (2002) summarized these theories and reviewed the intended and unintended consequences of 
using incentives in surveys. Some of this literature is reviewed briefly below, focusing primarily on topics 
relevant to incentives in telephone surveys. 

A rneta- analysis by Church (1993) shows that the payment of incentives in mail surveys is one 
method that consistently results in higher response rates. A similar finding was reported in a study by 
Trussell and Lavrakas (2004). They illustrated that for households that did not have a prior agreement to 
complete a survey, incremental increases in cash incentives were associated with increased response. 
Shettle and Mooney (1999) reported on an experiment in using monetary incentives that further supports 
the cost-effectiveness of incentives in mail surveys that are conducted under the aegis of the U.S. Federal 
Government. The studies of incentives nearly always find that prepaid incentives (those offered without 
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requiring the recipient to respond) are more effective than promised incentives that are contingent on 
response. Berk et al. (1987) found that for an in-person survey prepaid incentives were effective, but 
promised incentives were not. Church (1993) reported a similar finding for mail surveys. Singer et al. 
(1999) and Gelman, Stevens, and Chan (2003) use different meta-analysis models for telephone surveys 
and do not agree that promised incentives are ineffective, but do agree that prepaid incentives are more 
effective than promised ones. 

The evidence on the utility of incentives in RDD surveys is more limited and more recent. One of 
the reasons for the delay in the use of incentives in telephone surveys is that it is more difficult to send 
prepaid incentives because addresses cannot be obtained for some sampled telephone numbers. Recently, 
the proportion of residential telephone numbers that can be linked to addresses using inexpensive 
methods has increased. This development has made sending advance mailings and prepaid incentives 
more attractive for RDD surveys. Both Singer et al. (1999) and Gelman, Stevens, and Chan (2003) have 
found that monetary incentives increase response rates in RDD surveys. 

Another way of using incentives is to provide an incentive only to those units that refuse to 
complete the interview in the initial attempt. However, the published literature on incentives for refusal 
conversion is limited. Martin, Abreu, and Winters (2001) found that response rates increased when 
monetary incentives were used to convert refusers in the Survey of Income and Program Participation, a 
Census Bureau survey. Groves et al. (1999) studied the effect of the differential use of incentives (i.e., 
only for refusal conversion) and found that most respondents already believed incentives were commonly 
used by survey organizations. Knowledge of the fact that other respondents were given incentives or 
different levels of incentives did not have a statistically significant effect on the participation of sampled 
persons for future surveys. Less is known about the effectiveness of incentives at the refusal conversion 
stage for RDD surveys. In one published report addressing this issue, Singer, van Hoewyk, and Maher 
(2000) found that prepaying incentives was effective in increasing survey participation among those who 
had initially refused to participate. 

In addition to the initial contact with an adult in a sampled household, called the Screener 
interview in NHES, obtaining the cooperation of persons sampled in the Screener for extended interviews 
is very important. However, no reports on the effectiveness of incentives for improving extended 
interview response rates exist. Generally, those sampled for extended interviews for NHES topical 
surveys have been willing to participate at relatively high levels, especially when a parent is responding 
about a sampled child. Response rates for surveys of adults about their adult education experiences have 
traditionally been lower. For example, in NHES:2001, the weighted unit response rate for the surveys of 
parents about their children was 87 percent while the corresponding adult education survey unit response 
rate was 77 percent (these are rates computed for sampled persons in households that completed the 
Screener). As a result, there was much interest in the potential of incentives to increase the response rates 
for the AEWR-NHES:2003 and a test of incentives for this survey was included in the experiment. 



3. Screener Experiment 

The overall objective of the incentive experiment in 2003 was to determine if there were 
economical alternatives for improving unit response rates in NHES. In light of the results of studies 
comparing the effectiveness of prepaid and promised incentives (see section 2), only prepaid incentives 
were used in this experiment. The experiment focused primarily on the Screener because this is where 
most nonresponse occurs, but an additional experiment for the adult education survey was also included. 
The experiment at the Screener level examined the effect of different treatments in the initial contact, at 
the first refusal conversion stage, and at the second refusal conversion stage. The adult education survey 
experiment (discussed in the next section) was only at the first refusal conversion stage. 



6 




Monetary Incentives and Mailing Procedures 



Since advance first class letters had been shown to be effective in improving response rates in 
previous administrations of the NHES, a first class letter in a U.S. Department of Education envelope and 
on official stationery was sent to every sampled telephone number for which an address could be 
identified. Alternatives such as sending the advance letter by Priority Mail were considered to be too 
costly. 



The treatments for the Screener that were subject to manipulation were 

■ the amount of money included with the advance letter; 

■ the type of mail used for converting those that refused the initial contact; and, 

■ the amount of money included with the refusal mailing. 

In addition, mailing an official NHES brochure has been considered a possible method of 
improving response rates. To test this possibility, one of the experimental groups that received no 
monetary incentive was sent a brochure in the initial mailing. 

Two types of mailings were used in this experiment to examine their effects on refusal conversion; 
these were first class mail sent in U. S. Department of Education letterhead envelopes (with a mailing cost 
of $0.37 per letter), and Priority Mail sent in U. S. Postal Service Priority Mail envelopes (with a mailing 
cost of $3.85 per letter). The experimental groups were developed to permit the examination of the 
following effects on response rates: 

■ the effect of varied incentives in the advance letter ($0, $2, and $5); 

■ the effect of varied incentives in the refusal conversion letter ($0, $2, and $5); and 

■ the effect of Priority Mail versus first class mail for the refusal conversion letter. 

Table 2 shows the 10 experimental conditions included in the Screener experiment. The sample 
sizes for each of the conditions and for the nonmailable 3 cases (which were excluded from the 
experiment) are also given in the table. Addresses were identified for 59,365 of the telephone numbers in 
the experiment. Because the experimental treatments all involve mailings, telephone numbers for which 
an address could not be found were not included in the experiment. The sample also included 50,435 
telephone numbers for which addresses could not be identified. The percentage of cases for the 
experiment is affected by differential sampling of cases with addresses and differential sampling of 
telephone numbers in areas with a high concentration of minorities. Hagedorn et al. (forthcoming) 
provide the details on the sample design for NHES:2003. The fact that all numbers cannot be treated will 

depress the effectiveness of any strategy that involves mailing letters, but the effect is not as great as 

might be expected. In NHES:2003, 90 percent of all numbers that could be determined to be residential 
numbers had addresses (85 percent when the numbers are appropriately weighted by the inverse of the 
probability of selection). For most of this report, the focus is on the cases with addresses that were 
included in the experiment. Where relevant, cases without addresses are included in tabulations. 



’Nonmailable cases are the cases for which an address could not be obtained. These cases were not sent an advance letter or incentive. This 
subgroup was not treated as a control group for this experiment because there is a considerable literature, including results from previous NHES 
surveys, that shows that this subgroup has lower response rates in RDD surveys than mailable cases. 
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Table 2. Sample sizes for Screener experimental conditions: 2003 


Group 


Advance letter 


Refusal conversion letter 
plus brochures 


Sample 

size 1 


Mailing method 


Incentive 


Mailing method 


Incentive 


Total 


f 


f 


f 


f 


109,800 


Total mailable cases 


f 


f 


f 


f 


59,365 


1 


First class/brochure 


$0 


First class 


$0 


5,765 


2 


First class 


$0 


First class 


$2 


5,700 


3 


First class 


$0 


First class 


$5 


5,700 


4 


First class 


$0 


Priority 


$0 


6,850 


5 


First class 


$0 


Priority 


$2 


5,700 


6 


First class 


$2 


First class 


$0 


6,850 


7 


First class 


$2 


Priority 


$0 


5,700 


8 


First class 


$2 


Priority 


$2 


5,700 


9 


First class 


$5 


First class 


$0 


5,700 


10 


First class 


$2 


First class 


$2 


5,700 


Total nonmailable cases 


f 


f 


f 


f 


50,435 



t Not applicable. 

1 For mailable cases, the sample size is the number of telephone numbers for which an address match was obtained, randomly 
allocated to the experimental conditions. For nonmailable cases, the sample size is the number of telephone numbers. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 
Program, 2003. 



Group 1 is essentially a control group, but every case in this group contained a treatment that was 
not included in any other group. Along with the advance letter, the group 1 cases were sent a short color 
brochure describing NHES to determine the possible effectiveness of a brochure for increasing response 
rates. The brochure was not included in any other advance mailings. Thus, the experimental design allows 
for a test of the effect of the brochure in the $0 advance incentive condition on whether potential 
participants will ever refuse to respond to the Screener survey. A brochure was included with an 
informational letter if a respondent specifically asked to be mailed more information on the study before 
agreeing to be interviewed. Refusal conversion mailings included brochures. 

The sample sizes in the table arc the number of telephone numbers with addresses assigned to 
each group. Group 4 (no incentive in either the advance letter or the refusal conversion letter, and a 
refusal conversion letter sent by Priority Mail) is similar to the condition that was used in NHES:2001, 
although in 2001, FedEx was used rather than Priority Mail. 4 Larger sample sizes were assigned to groups 
4 and 6 because they were identified as the treatments that might increase response rates and be 
acceptable approaches in a federal government survey in that advance incentives have some advantages 
over refusal conversion incentives. 

Once sample sizes for the Screener experimental groups were determined and the cases were 
randomly assigned to the groups, advance letters were mailed with or without incentives according to the 



4 In NHES:2001, FedEx was used for refusal letters, with the exception of rural route and Post Office box addresses, to which FedEx cannot be 
sent. For these addresses. Priority Mail was used. 
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plan. If the household refused to participate, refusal conversion letters were sent with the appropriate 
incentives. If a refusal was hostile (profane or abusive), no further efforts were made to contact the 
household. In addition, a small number of cases that directly contacted the U.S. Department of Education 
to refuse received no further contact. 

For analyses of the effects of the treatments on refusal conversion, the sample sizes in each group 
arc considerably smaller, because only the subset of the telephone numbers with addresses for which 
Screener refusal conversion is necessary are included. Of the 59,365 telephone numbers in the 
experiment, only 20,259 cases required and received Screener refusal conversion letters (see table 10 for 
the total number of cases in each incentive group that ever refused). 

If the household refused a second time, they were subject to a second refusal conversion call. To 
boost response rates, a sample of 75 percent of the households that had not received a Priority Mail letter 
previously (groups 1, 2, 3, 6, 9, and 10) was sent a Priority Mail letter prior to the second refusal 
conversion call. Only 75 percent were sent by Priority Mail so that the effectiveness of the special mailing 
could be evaluated. The sampling to determine which numbers were assigned to the special mailing was 
done randomly within the experimental groups. The findings section of this report focuses on the initial 
and first refusal conversion rates to avoid the effect of the Priority Mailing at second refusal. The results 
of the Priority Mail at the second refusal conversion arc then analyzed separately. 

An examination of the treatments applied to each experimental group (shown in table 2) reveals 
that a full factorial design was not used for this experiment. One reason for this is that, while testing each 
level of advance incentive ($0, $2, and $5), each level of refusal conversion incentive ($0, $2, and $5), 
and each mailing method for first refusal letters (first class and Priority Mail), the sample sizes in 
NHES:2003 were not sufficient to support a full factorial design. A second reason is that some treatments 
in a full factorial design (such as $5 advance incentive, $5 refusal conversion incentive, and Priority Mail 
for the refusal mailing) were deemed cost-prohibitive for NHES; therefore, even if such a treatment was 
found to be effective, there is little chance that it would ever be used in NHES. A third reason is that by 
balancing the experimental sample with respect to prior treatments or conditioning on the prior treatments 
during analysis, it is possible to minimize confounding effects. Exhibit 1 summarizes the key 
experimental comparisons that can be made with respect to the Screener treatments used in the 
experiment. 

Table 3 shows the final disposition code for all the telephone numbers included in the experiment. 
In addition, the numbers without addresses are included for comparison. This table gives an overall 
assessment of the effects of the various treatment groups. There are essentially five dispositions: an adult 
in the household completed the Screener interview, the number was determined to be nonresidential, all 
the calls to the household resulted in either a ring no-answer or an answering machine, the household 
refused to participate, or the household did not respond for some other reason (such as a language barrier 
or it was not available during the field period). The percentages in the table arc weighted by the 
household base weight to take account of the differential sampling of minority areas and telephone 
numbers with addresses. All other analyses in this paper also use the household base weights. 

Table 4 gives the mean number of calls required to finalize the experimental Screener cases. This 
table gives a measure of the workload required on the paid of the interviewer in order to finalize a 
Screener. On average, it required seven calls to finalize a Screener. This mean includes calls made to 
telephone numbers that were not resolved as residential or nonresidential. The table also gives the mean 
numbers of call attempts by final disposition code of the Screener. There is significant valuation in the 
mean number of call attempts to finalize a Screener, ranging from three calls for nonresidential numbers 
to 22 calls for never-answered numbers. For nonmailable cases, an average of six calls were needed to 
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finalize a Screener, but nearly 60 percent of these numbers were identified as nonworking or 
nonresidential before any call attempts were made by the interviewers. 

Exhibit 1. Summary of major study comparisons 



Experimental 

process 


Characteristic(s) 


Aspect(s) compared 


Groups compared 


Overall 


Unit response rate 


No monetary incentive vs. 
any monetary incentive 


Groups 1, 4 vs. groups 2, 3, 5, 
6,7,8,9,10 


Advance 

mailing 


Ever refused rate 


Brochure vs. no brochure 


Group 1 vs. groups 2, 3, 4, 5 


Advance 

mailing 


Ever refused rate 


$0 advance incentive vs. $2 
advance incentive vs. $5 
advance incentive 


Groups 1, 2, 3, 4, 5 vs. groups 
6, 7, 8, 10 vs. group 9 


Advance 

mailing 


Percent completed for 
cases that never refused 


$0 advance incentive vs. $2 
advance incentive vs. $5 
advance incentive 


Groups 1, 2, 3, 4, 5 vs. groups 
6, 7, 8, 10 vs. group 9 


First refusal 
mailing 


First refusal conversion 
rate 


$0 refusal conversion 
incentive vs. $2 refusal 
conversion incentive 1 


Groups 1, 4, 6, 7 vs. groups 2, 
5, 8, 10 


First refusal 
mailing 


Unit response rate after 
first refusal conversion, 2 
first refusal conversion 
rate 


Priority Mail vs. first class 
for first refusal mailing, for 
groups with the same 
advance and refusal 
conversion monetary 
incentives 


Group 1 vs. group 4; group 2 
vs. group 5; group 6 vs. group 
7; group 8 vs. group 10 


Second 

refusal 

mailing 


Percent completed after 
second refusal conversion 


Priority Mail vs. no mailing 
for second refusal 
conversion 


Group 1 Priority Mail vs. 
Group 1 no mailing, and 
similar comparisons for 
groups 2, 3, 6, 9, 10 



1 The balance of the design allows for first refusal conversion comparisons without considering the advanced incentive, since half 
of the four groups with an initial incentive of SO had a $2 conversion incentive and half of the four groups with an initial incentive 
of $2 had a $2 conversion incentive, and the sample sizes in each group were approximately equal. 

2 Screeners that were converted on second refusal conversion attempts are counted as nonrespondents in this comparison. 



A primary goal of the Screener experiment was to determine if the different incentive and mailing 
conditions would have variable effects on response rates. Therefore, much of the analysis focuses on 
whether interviews were completed and related estimates. One of the statistics used is the unit response 
rate. The definition of the unit response rate used is explained in detail in appendix A, but is briefly 
described here. The overall unit response rate is the percentage of interviews completed taking all survey 
stages into account. For the Screener interview, there is only one survey stage. To compute the Screener 
response rate, it is assumed that 19.7 percent of the sampled numbers classified as never answered were 
residential. 
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Table 3. Weighted percentage distributions of final disposition codes for numbers sampled for the Screener, by incentive group: 2003 



Incentive group 


Sample 

size 1 


Final disposition 


Percent complete 


Percent 

nonresidential 


Percent 
never answered 


Percent refused 


Percent other 
nonresponse 2 


Total 


109,800 




— 




— 




— 




— 




— 


Total mailable cases 


59,365 


53.3 


(0.20) 


21.2 


(0.19) 


2.0 


(0.04) 


17.9 


(0.17) 


5.6 


(0.10) 


1 - ($0 brochure/lst $0).. 


5,765 


50.5 


(0.65) 


21.3 


(0.55) 


2.2 


(0.14) 


20.4 


(0.51) 


5.7 


(0.36) 


2 -($0/1 st $2) 


5,700 


53.0 


(0.63) 


21.2 


(0.51) 


2.0 


(0.14) 


17.8 


(0.53) 


5.9 


(0.34) 


3 -($0/1 st $5) 


5,700 


54.7 


(0.62) 


21.3 


(0.57) 


2.0 


(0.13) 


16.5 


(0.49) 


5.5 


(0.24) 


4 - ($0/Priority $0) 


6,850 


50.0 


(0.64) 


21.4 


(0.54) 


2.1 


(0.13) 


20.7 


(0.54) 


5.8 


(0.30) 


5 - ($0/Priority $2) 


5,700 


52.7 


(0.68) 


21.0 


(0.59) 


2.0 


(0.13) 


18.3 


(0.54) 


6.0 


(0.28) 


6 - ($2/1 st $0) 


6,850 


53.2 


(0.54) 


21.6 


(0.50) 


2.0 


(0.10) 


17.6 


(0.46) 


5.6 


(0.29) 


7 - ($2/Priority $0) 


5,700 


54.8 


(0.78) 


20.5 


(0.68) 


1.9 


(0.12) 


17.3 


(0.51) 


5.5 


(0.31) 


8 - ($2/Priority $2) 


5,700 


54.2 


(0.75) 


21.6 


(0.61) 


1.8 


(0.11) 


17.1 


(0.59) 


5.3 


(0.29) 


9 - ($5/1 st $0) 


5,700 


54.8 


(0.60) 


21.3 


(0.52) 


1.9 


(0.15) 


16.6 


(0.57) 


5.3 


(0.31) 


10 -($2/ 1st $2) 


5,700 


55.2 


(0.58) 


21.0 


(0.61) 


2.1 


(0.12) 


16.6 


(0.47) 


5.1 


(0.31) 


Total nonmailable cases 


50,435 


6.0 


(0.12) 


87.8 


(0.15) 


1.7 


(0.03) 


3.5 


(0.09) 


1.0 


(0.04) 



— Not reported. 

1 For mailable cases, the sample size is the number of telephone numbers for which an address match was obtained, randomly allocated to the experimental conditions. For 
nonmailable cases, the sample size is the total number of telephone numbers; this total includes 30,725 telephone numbers that were identified as business or nonworking numbers 
through pre-field procedures and were never dialed. 

2 Other nonresponse includes cases that finalized due to language problems, the inability to make contact with the respondent (maximum call status), or the respondent being 
unavailable during the field period. 

NOTE: Standard error shown in parenthesis. Percents may not sum to 100 because of rounding. The parenthetical descriptions of the experimental conditions give the advance 
mailing condition (before the / mark) and the initial refusal condition (after the / mark). All advance letters were sent by first class mail in a U.S. Department of Education business 
envelope, as were initial refusal letters in conditions not specifying Priority Mail. The Priority Mail letters were sent in the U.S. Postal Service Priority Mail envelopes. Refusal 
letters included a colored NF1ES project brochure. Final dispositions include the results of the second refusal conversion attempts. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Flousehold Education Surveys Program, 2003. 
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Table 4. Mean number of Screener call attempts for mailable cases and for nonmailable cases, by 
final disposition: 2003 



Final Screener disposition 


Sample size 1 


Mean screening call attempts 


Total 


79,075 




— 


Total mailable cases 


59,365 


7.0 


(0.04) 


Completed 


29,196 


4.7 


(0.03) 


Nonresidential 


12,259 


3.4 


(0.04) 


Never answered 


3,385 


22.3 


(0.10) 


Refused 


10,102 


12.9 


(0.09) 


Other nonresponse 2 


4,423 


18.7 


(0.18) 


Total nonmailable cases 


19,710 


5.9 


(0.05) 


Completed 


2,853 


5.5 


(0.09) 


Nonresidential 


9,968 


3.2 


(0.03) 


Never answered 


4,730 


16.8 


(0.07) 


Refused 


1,659 


13.8 


(0.18) 


Other nonresponse 2 


500 


19.9 


(0.41) 



— Not reported. 

1 For mailable cases, the sample size is the number of telephone numbers for which an address match was obtained, randomly 
allocated to the experimental conditions. For nonmailable cases, the sample size of 19,710 is the total number of telephone 
numbers that were ever dialed; this total excludes 30,725 telephone numbers that were identified as business or nonworking 
numbers through pre-field procedures and were never dialed. These pre-field procedures only identify a subset of business and 
non working numbers. 

2 Other nonresponse includes cases that finalized due to language problems, the inability to make contact with the respondent 
(maximum call status), or the respondent being unavailable during the field period. 

NOTE: Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 
Program, 2003. 



For some analyses the unit response rates are not the most relevant measures. A reason noted 
earlier for focusing on other statistics, in addition to response rates, is that the response rates include the 
effect of the Priority Mailing at second refusal conversion. This mailing was differential and not paid of 
the main Screener incentive experiment. It only affected groups without any previous Priority Mailing 
(groups 1, 2, 3, 6, 9, and 10). A second reason that other statistics may be more pertinent is that 
experimental groups (e.g., all groups sent a $2 incentive in the advance mailing) are sometimes combined 
to increase the precision of the estimates of the effect of the treatment. Because the combined groups have 
different treatments at different stages of refusal conversion, the response rates for the groups confound 
the treatment effects. 

Instead of response rates, either “ever refusal” rates or the percentage of cases that complete the 
interview at a specific stage of the process are often provided. Ever refusal rates arc used primarily to 
evaluate the effect of a treatment in the first stage, prior to any refusal conversion. Ever refusal rates give 
the (weighted) number of cases in the group that refused the Screener divided by the total weighted 
number of cases that either completed or refused the Screener. The nonresponse cases that were never 
classified as refusals (such as those with language problems or that arc not available in the field period) 
are not included in the computations of this rate. The ever refusal rate is the complement of the rate that 



12 




Monetary Incentives and Mailing Procedures 



AAPOR (2004) calls the cooperation rate. The term “ever refusal rate” is preferred because cooperation 
and response rates are often confused. 

In addition to measures related to improving response rates, estimates of the number of call 
attempts required to finalize the case or to complete a specific phase of the interview are produced. For 
example, table 4 provides the mean numbers of call attempts by the final disposition of the Screener. In 
other tables, statistics such as the mean numbers of call attempts needed to move from an initial refusal to 
a second refusal by experimental group arc given. These numbers are used as an indication of whether 
there are obvious cost issues associated with an experimental group beyond the direct costs of the 
mailings and incentives. 

The results from the Screener experiment are given below, and the adult education survey 
experiment is discussed in the next section. All of the analyses were done with the appropriate survey 
weights using WesVar software (Westat 2000) to calculate jackknife variance estimates. Tests of 
hypotheses were done using two-tailed t-tests with a 5 percent type I error rate. Bonferroni adjustments 
were used to compensate for multiple comparisons to ensure that the overall significance level associated 
with all of the comparisons remained at or below the specified level of 5 percent. Given the problem of 
falling response rates, relatively small increases in response rates associated with a particular 
experimental condition are considered noteworthy in this paper. Differences that are not statistically 
significant and statistically significant differences of less than one percentage point are not discussed. 

Table 5 gives the Screener response rates and ever refusal rates for each of the 10 treatment 
groups. Since nonresidential telephone numbers arc not used in computing response rates, the 30,725 
nonresidential numbers identified through prefield procedures, the 12,259 nonresidential mailable 
numbers, and the 9,968 nonresidential nonmailable numbers (see table 4) are excluded, and the estimates 
in the table are based on the remaining 47,106 mailable and 9,742 nonmailable numbers. The 
corresponding rates for the cases without addresses are also presented for comparison purposes. The 
response rates in this table and the percentage of sampled cases that were completed given in table 3 
highlight the efficiency considerations that led to oversampling telephone numbers with addresses in the 
survey. Figure 2 contains a graphical display of the Screener response rates and ever refusal rates by 
incentive group. 



Advance Mailing 

The first topic considered is the effect of including the brochure in the advance mailing. This is 
the stalling point because if the brochure does improve cooperation, then it would not be useful to 
combine group 1 with other groups that have no monetary incentive in the advance mailing. If including 
the brochure were effective, the ever refusal rate for group 1 should be lower than the ever refusal rates 
for the other groups with no money in the advance letter (groups 2, 3, 4, and 5). The results in table 6 
indicate that the inclusion of the brochure in the advance letter does not increase cooperation at the initial 
stage of the process. In fact, the initial cooperation rate for the brochure group (group 1) was lower than 
the rate for the other groups with no advance incentive combined. 

Table 5 illustrates that the ever refusal rates show the type of pattern expected, with rates for those 
with no incentive higher than those with $2 or $5 in the advance letter. The size of the differences by 
incentive amount is discussed in connection with the next table (table 6). Table 5 also gives the response 
rates for the 10 experimental groups. Keeping in mind the caveat given earlier about the potential effects 
of the second refusal conversion Priority Mailing (sent to subsamples of groups 1, 2, 3, 6, 9, and 10), 
these rates are examined. The unit response rates for the two groups with no incentive (groups 1 and 4) 
are generally lower than the rates for the other groups. Ordering the groups from those sent no monetary 
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incentive to those sent either $4 or $5 (groups 8, 9, and 10), it can be seen that the higher incentives do 
result in higher response rates (table 5), but with diminishing returns. Figure 3 illustrates this trend. The 
effect of the refusal conversion mailings is discussed later. 



Table 5. Screener response rates and ever refusal rates, by incentive group: 2003 



Incentive group 


Sample size 1 


Response rate, percent 


Ever refused rate, percent 


Total 


56,848 




— 




— 


Total mailable cases 


47,106 




— 




— 


1 - ($0 brochure/ 1st $0) 


4,574 


64.1 


(0.67) 


48.8 


(0.76) 


2 - ($0 /1st $2) 


4,524 


67.3 


(0.71) 


47.1 


(0.84) 


3 - ($0 /1st $5) 


4,518 


69.5 


(0.64) 


45.4 


(0.74) 


4 - ($0 /Priority $0) 


5,422 


63.7 


(0.69) 


46.6 


(0.69) 


5 - ($0 /Priority $2) 


4,543 


66.7 


(0.71) 


45.2 


(0.83) 


6 -($2 /1st $0) 


5,424 


67.9 


(0.67) 


40.9 


(0.67) 


7 - ($2 /Priority $0) 


4,558 


68.9 


(0.72) 


42.1 


(0.69) 


8 - ($2 /Priority $2) 


4,498 


69.1 


(0.76) 


42.4 


(0.81) 


9 - ($5 /1st $0) 


4,505 


69.7 


(0.70) 


38.9 


(0.76) 


10 -($2 /1st $2) 


4,540 


69.9 


(0.56) 


42.1 


(0.70) 


Total nonmailable cases 


9,742 


49.3 


(0.59) 


55.9 


(0.64) 



— Not reported. 

1 The sample size is the number of sampled telephone numbers excluding those numbers that were classified as nonresidential. 
NOTE: Standard error shown in parenthesis. The parenthetical descriptions of the experimental conditions give the advance 
mailing condition (before the / mark) and the initial refusal condition (after the / mark). All advance letters were sent by first 
class mail in a U.S. Department of Education business envelope, as were initial refusal letters in conditions not specifying 
Priority Mail. The letters sent by Priority Mail were sent in the U.S. Postal Service Priority Mail envelopes. Refusal letters 
included a colored NHES project brochure. Nonresidential telephone numbers are excluded from the table. The sample sizes are 
unweighted. The rates are calculated after weighting cases in the sample for differential probabilities of selection; therefore, rates 
cannot be calculated directly from the sample sizes presented. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 
Program, 2003. 
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Figure 2. Screener response rates and ever refusal rates, by incentive group: 2003 



Incentive group 

1 - ($0 brochure/ 
1st $0) 

2 - ($0/1 st $2) 

3 - ($0/1 st $5) 
4 - ($0/Priority $0) 



5 - ($0/Priority $2) 



6 - ($2/ 1st $0) 



7 - ($2/Priority $0) 



8 - ($2/Priority $2) 



9 - ($5/ 1st $0) 
10 - ($2/ 1st $2) 




■ Unit response 
rate (%) 

□ Ever refused 
rate (%) 



0 35 40 45 50 55 60 65 70 75 



Rate (%) 



NOTE: The incentive group descriptions give the advance mailing condition before the / mark and the initial refusal condition 
after the / mark. After the / mark, I s1 means the initial refusal materials were sent by first class mail. After the / mark, Priority 
means the initial refusal materials were sent by Priority Mail. All advance letters were sent by first class mail in a U.S. 
Department of Education business envelope, as were initial refusal letters in conditions specifying first class mail. The letters sent 
by Priority Mail were sent in the U.S. Postal Service Priority Mail envelopes. Refusal letters included a colored NHES project 
brochure. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 
Program, 2003. 
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Table 6 combines groups with a similar advance mailing treatment, group 1 is included with the 
other groups with no incentive. The ever refusal rates are the most important rates in this table since the 
response rates combine groups with different subsequent treatments. The ever refusal rates for the three 
groups differ, with the highest rates associated with no incentive and the lowest rates for the $5 incentive 
group. 1 One way of thinking about the relationship is that adding the $2 incentive reduces the ever refusal 
rate by 2.4 percentage points for each dollar relative to no incentive, while adding $5 reduces the refusal 
rate by 1.5 percentage points for each dollar relative to no incentive. Thus, while the larger incentive is 
more effective in the experiment, the relative effectiveness decreases when more money is sent, at least 
for the dollar values considered here. 



Table 6. Screener response rates and ever refusal rates, by advance letter incentive: 2003 



Advance letter 
incentive 


Sample size 1 


Response rate, percent 


Ever refused rate, percent 


$0 


23,581 


66.1 (0.31) 


46.6 (0.30) 


$2 


19,020 


68.9 (0.36) 


41.8 (0.37) 


$5 


4,505 


69.7 (0.70) 


38.9 (0.76) 



NOTE: Standard error shown in parenthesis. The sample sizes are unweighted. The rates are calculated after weighting cases in 
the sample for differential probabilities of selection; therefore, rates cannot be calculated directly from the sample sizes 
presented. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 
Program, 2003. 



Refusal Conversion 

In the previous section, advance mailing conditions were evaluated. Here, treatments involving 
experimental conditions for Screener refusal conversion are considered. These treatments (refusal 
conversion mailings and possible further monetary incentives) were applied only if a household refused to 
complete the Screener. Before discussing the effects of the refusal conversion treatments, it is important 
to remember that most households that completed the Screener never refused (table 7). To help put refusal 
conversion strategies into perspective, analysis is first conducted to show how cases that never refused 
behaved by the kind of incentive they received in the advance mailing (table 8). Next, cases that ever 
refused are studied (table 9). It should be kept in mind that cases that ever refused include those that 
refused more than once. 



1 The ever refusal rate higher for the brochure group than 2 of the other 4 groups that received no initial monetary incentives. However, all groups 
that received some initial monetary incentive (6-10) had lower ever refusal rates than all groups that received no initial monetary incentives (1-5). 
For this reason, groups that received no initial monetary incentives were combined to facilitate comparisons. 
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Figure 3. Screener response rates by total incentive amount and incentive group: 2003 




0 35 40 45 50 55 60 65 70 75 



Unit response rate (%) 



NOTE: All advance letters were sent by first class mail in a U.S. Department of Education business envelope, as were initial 
refusal letters in conditions not specifying Priority Mail. The letters sent by Priority Mail were sent in the U.S. Postal Service 
Priority Mail envelopes. Refusal letters included a colored NHES project brochure. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 
Program, 2003. 



17 



Monetary Incentives and Mailing Procedures 



Table 7. Percentage distribution of Screener completes for mailable cases that completed the 
Screener, by refusal conversion stage: 2003 




Total 29,196 100.0 (f) 



Total 29,196 100.0 (f) 

Never refused 20,501 70.3 (0.28) 

First refusal conversion 6,740 23.1 (0.27) 

Second refusal conversion 1,955 6.7 (0.14) 

t Not applicable. 



1 The sample size is the number of mailable telephone numbers with completed Screeners. 

NOTE: Nonresidential telephone numbers are excluded from the table. Percents may not sum to 100 because of rounding. The 
sample sizes are unweighted. The percentages are calculated after weighting cases in the sample for differential probabilities of 
selection; therefore, percentages cannot be calculated directly from the sample sizes presented. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 
Program, 2003. 



Table 8 shows that, among cases that never refused to complete the Screener, the percent that 
completed the Screener was lower for cases that received no incentive in the advance mailing than it was 
for cases that did. The cases that did not complete the Screener could be nonresidential cases that never 
answered the phone, or other nonresponse. 2 



Table 8. Percentage of cases with completed Screeners that never refused, by advance letter 
incentive: 2003 



Advance letter Sample Mean calls to 

incentive size 1 Percent completed Mean total calls 2 complete 2 

$0 18,927 57.0 (0.37) 5.5 (0.06) 3.8 (0.05) 

$2 16,158 60.2 (0.41) 5.2 (0.05) 3.7 (0.04) 

$5 3,955 61.7 (0.73) 4.9 (0.12) 3.6 (0.11) 



Sample size is the number of telephone numbers at which a refusal was never received, minus numbers identified as 
nonresidential prior to calling (i.e., through prefield activities). 

“ The total ccdls estimates include calls to all telephone numbers in the subgroup, while the calls to complete estimates include 
calls to only the telephone numbers for which completed Screeners were obtained. 

NOTE: Standard error shown in parenthesis. The SO advance letter incentive category combines incentive groups 1. 2, 3, 4, and 5. 
The $2 advance letter incentive category combines incentive groups 6, 7, 8, and 10. The $5 advance letter incentive category is 
incentive group 9. The sample sizes are unweighted. The percentages are calculated after weighting cases in the sample for 
differential probabilities of selection; therefore, percentages cannot be calculated directly from the sample sizes presented. 
SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 
Program, 2003. 



2 There were 16,73 1 nonmailable cases that never refused and 15.6 percent of these completed the Screener (s.e. = 0.32 percent). The mean 
numbers of calls for these cases were 4.6 for all cases and 4.3 for completed cases (with standard errors of 0.04 and 0.1 1, respectively). 
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Table 9 includes all cases that ever refused to complete the Screener; thus, it excludes all the cases 
shown in table 8. The percentage completed (those cases that eventually completed a Screener) is an 
overall measure of the joint effectiveness of the first and second refusal conversion attempts. Several of 
the groups with larger completion rates in the table are those that had Priority Mailings that were sent to 
75 percent of the second refusals (groups 1, 2, 3, 6, 9, and 10 [data not shown]). Because of the 
confounding of the two refusal conversion treatments, these results are analyzed separately. 



Table 9. Percentage of cases that ever refused to complete the Screener and final disposition, by 
incentive group: 2003 



Incentive group 


Sample size 1 


Final disposition 


Percent completed 


Percent refused 


Total 


23,230 




— 




— 


Total mailable cases 


20,259 




— 




— 


1 - ($0 brochure/ 1st $0) 


2,166 


42.9 


(1.17) 


50.6 


(1.05) 


2 - ($0 /1st $2) 


2,088 


47.2 


(1.14) 


45.6 


(1.08) 


3 - ($0 /1st $5) 


2,023 


49.1 


(1.16) 


43.8 


(1.15) 


4 - ($0 /Priority $0) 


2,486 


39.5 


(1.17) 


53.7 


(1.16) 


5 - ($0 /Priority $2) 


2,007 


44.2 


(1.16) 


48.8 


(1.18) 


6 -($2 /1st $0) 


2,188 


40.0 


(1.07) 


52.0 


(1.08) 


7 - ($2 /Priority $0) 


1,850 


44.3 


(1.25) 


49.7 


(1.25) 


8 - ($2 /Priority $2) 


1,839 


44.8 


(1.28) 


49.3 


(1.38) 


9 - ($5 /1st $0) 


1,734 


40.8 


(1.32) 


51.7 


(1.25) 


10 -($2 /1st $2) 


1,878 


45.1 


(1.08) 


47.3 


(1.23) 


Total nonmailable cases 


2,971 


28.2 


(0.89) 


55.9 


(1.02) 



— Not reported. 

1 Sample size is the number of telephone numbers at which a refusal was received. 

NOTE: Standard error shown in parenthesis. The parenthetical descriptions of the experimental conditions give the advance 
mailing condition (before the / mark) and the initial refusal condition (after the / mark). All advance letters were sent by first 
class mail in a U.S. Department of Education business envelope, as were initial refusal letters in conditions not specifying 
Priority Mail. The letters sent by Priority Mail were sent in the U.S. Postal Service Priority Mail envelopes. Refusal letters 
included a colored NHES project brochure. Details do not sum to 100 percent because of the exclusion of the “Other 
nonresponse” category. The sample sizes are unweighted. The percentages are calculated after weighting cases in the sample for 
differential probabilities of selection; therefore, percentages cannot be calculated directly from the sample sizes presented. 
SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 
Program, 2003. 



Table 10 gives the mean number of calls needed to reach the final disposition status for Screener 
refusal conversion cases by the experimental groups. The values in the table show little variation by 
experimental group in the mean total calls, the mean calls to obtain a completed Screener, and the mean 
calls to obtain a final refusal. Table 1 1 gives more details on the mean number of call attempts, dividing 
the total call attempts into those used to obtain the first refusal and those used to finalize the initial refusal 
case as either a completed Screener or final refusal. Even with the added detail, the mean numbers of calls 
generally did not vary across experimental groups. 
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Table 10. Mean number of calls to final status for Screeners that ever refused, by incentive group: 
2003 



Incentive group 


Sample 

size 1 


Mean total calls 


Mean calls to 
complete 


Mean calls to 
refusal 


1 - ($0 brochure /1st $0) 


2,166 


10.1 


(0.17) 


7.4 


(0.18) 


12.5 


(0.24) 


2 - ($0 /1st $2) 


2,088 


10.0 


(0.20) 


7.1 


(0.20) 


13.0 


(0.31) 


3 - ($0 /1st $5) 


2,023 


9.8 


(0.18) 


6.8 


(0.17) 


13.2 


(0.26) 


4 - ($0 /Priority $0) 


2,486 


10.1 


(0.15) 


7.0 


(0.16) 


12.5 


(0.23) 


5 - ($0 /Priority $2) 


2,007 


10.3 


(0.20) 


7.2 


(0.18) 


13.1 


(0.32) 


6 -($2 /1st $0) 


2,188 


10.0 


(0.18) 


7.0 


(0.19) 


12.7 


(0.24) 


7 - ($2 /Priority $0) 


1,850 


10.0 


(0.19) 


7.1 


(0.18) 


12.6 


(0.29) 


8 - ($2 /Priority $2) 


1,839 


10.1 


(0.21) 


6.8 


(0.18) 


13.3 


(0.33) 


9 -($5 /1st $0) 


1,734 


10.5 


(0.18) 


7.0 


(0.22) 


13.4 


(0.28) 


10 -($2 /1st $2) 


1,878 


10.1 


(0.19) 


7.0 


(0.20) 


13.1 


(0.30) 



Sample size is the number of telephone numbers at which a refusal was received. 

NOTE: Standard error shown in parenthesis. The parenthetical descriptions of the experimental conditions give the advance 
mailing condition (before the / mark) and the initial refusal condition (after the / mark). All advance letters were sent by first 
class mail in a U.S. Department of Education business envelope, as were initial refusal letters in conditions not specifying 
Priority Mail. The letters sent by Priority Mail were sent in the U.S. Postal Service Priority Mail envelopes. Refusal letters 
included a colored NHES project brochure. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 
Program, 2003. 



Table 11. Mean number of calls for initial refusal Screener cases, by outcome of call and incentive 
group: 2003 







Mean number of calls 


Incentive group 


Sample 

size 1 


To initial refusal 


From refusal to 
complete 


From first to second 
refusal 



1 - ($0 brochure/ 1st $0) 2,166 3.9 



2 - ($0 /1st $2) 


2,088 


3.9 


3 - ($0 /1st $5) 


2,023 


3.8 


4 - ($0 /Priority $0) 


2,486 


3.8 


5 - ($0 /Priority $2) 


2,007 


4.0 


6 -($2 /1st $0) 


2,188 


3.7 


7 - ($2 /Priority $0) 


1,850 


3.9 


8 - ($2 /Priority $2) 


1,839 


4.0 


9 -($5 /1st $0) 


1,734 


3.9 


10 -($2 /1st $2) 


1,878 


4.0 



(0.10) 


4.1 


(0.12) 


3.3 


(0.10) 


(0.11) 


3.9 


(0.14) 


3.7 


(0.13) 


(0.10) 


3.7 


(0.12) 


3.5 


(0.11) 


(0.08) 


3.8 


(0.11) 


3.5 


(0.10) 


(0.12) 


4.0 


(0.13) 


3.6 


(0.12) 


(0.10) 


4.0 


(0.12) 


3.6 


(0.12) 


(0.10) 


3.9 


(0.14) 


3.7 


(0.13) 


(0.12) 


3.6 


(0.13) 


3.6 


(0.12) 


(0.10) 


3.9 


(0.16) 


3.7 


(0.14) 


(0.12) 


3.7 


(0.12) 


3.6 


(0.12) 



Sample size is the number of telephone numbers at which a refusal was received. 

NOTE: Standard error shown in parenthesis. The parenthetical descriptions of the experimental conditions give the advance 
mailing condition (before the / mark) and the initial refusal condition (after the / mark). All advance letters were sent by first 
class mail in a U.S. Department of Education business envelope, as were initial refusal letters in conditions not specifying 
Priority Mail. The letters sent by Priority Mail were sent in the U.S. Postal Service Priority Mail envelopes. Refusal letters 
included a colored NHES project brochure. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 
Program, 2003. 
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Table 12 shows the percentage of cases that refused that were completed after the first refusal 
efforts (called the first refusal conversion rate). These rates vary among the experimental groups from 29 
percent to 39 percent. If the use of Priority Mail at this stage is ignored, the 10 incentive groups can be 
classified into two groups that can be compared by the amount of incentive given in the conversion letter 
($0 or $2). The mean first conversion rate for those in the $0 group (groups 1, 4, 6. 7) is 31.8 percent (s.e. 
= 0.58 percent), and the mean for those in the $2 group (Groups 2, 5, 8. 10) is 35.7 percent (s.e. = 0.54 
percent). The groups with $2 conversion incentives have significantly higher first conversion rates than 
the groups with $0 conversion incentives (Groups 1. 4, 6, 7, and 9). While this analysis does not explicitly 
account for the incentive amount in the initial mailing, the balance of the design supports the conclusion 
that these results are the effect of the refusal conversion incentive amount. 3 “Balance” refers to the fact 
that, excluding incentive groups 3 and 9, half of the four groups with an initial incentive of $0 had a $2 
conversion incentive and half of the four groups with an initial incentive of $2 had a $2 conversion 
incentive. 

Table 12. Screener response rates after first conversion attempt, by incentive group: 2003 



Incentive group 


Response rate after first refusal 
conversion 1 


First refusal conversion rate 


Sample size 2 


Percent 


Sample size 3 


Percent 


Total 


56,848 




— 


23,230 




— 


Total mailable cases 


47,106 




— 


20,259 




— 


1 - ($0 brochure/ 1st $0) 


4,574 


57.9 


(0.78) 


2,166 


30.7 


(1.20) 


2 - ($0 /1st $2) 


4,524 


61.9 


(0.71) 


2,088 


36.3 


(1.05) 


3 - ($0 /1st $5) 


4,518 


64.6 


(0.76) 


2,023 


38.8 


(1.23) 


4 - ($0 /Priority $0) 


5,422 


59.9 


(0.70) 


2,486 


31.8 


(1.11) 


5 - ($0 /Priority $2) 


4,543 


62.4 


(0.68) 


2,007 


35.1 


(1.06) 


6 -($2 /1st $0) 


5,424 


63.4 


(0.69) 


2,188 


29.5 


(1.06) 


7 - ($2 /Priority $0) 


4,558 


65.0 


(0.78) 


1,850 


35.1 


(1.26) 


8 - ($2 /Priority $2) 


4,498 


65.6 


(0.73) 


1,839 


36.7 


(1.10) 


9 - ($5 /1st $0) 


4,505 


65.3 


(0.74) 


1,734 


29.9 


(1.30) 


10 -($2 /1st $2) 


4,540 


65.4 


(0.64) 


1,878 


34.9 


(1.10) 


Total nonmailable cases 


9,742 


45.6 


(0.63) 


2,971 


21.1 


(0.85) 



1 Screeners that were converted on second refusal conversion attempts are counted as nonrespondents in this table. 

2 For mailable cases, the sample size is the number of telephone numbers assigned to each experimental condition, minus 
numbers identified as nonresidential. For nonmailable cases, the sample size is the total number of telephone numbers, minus 
numbers identified as nonresidential. 

3 Sample size is the number of telephone numbers at which a refusal was received. 

NOTE: Standard error shown in parenthesis. The parenthetical descriptions of the experimental conditions give the advance 
mailing condition (before the / mark) and the initial refusal condition (after the / mark). All advance letters were sent by first 
class mail in a U.S. Department of Education business envelope, as were initial refusal letters in conditions not specifying 
Priority Mail. The letters sent by Priority Mail were sent in the U.S. Postal Service Priority Mail envelopes. Refusal letters 
included a colored NF1ES project brochure. The sample sizes are unweighted. The rates are calculated after weighting cases in the 
sample for differential probabilities of selection; therefore, rates cannot be calculated directly from the sample sizes presented. 
SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 
Program, 2003. 



1 A regression analysis, discussed in appendix B, also supports this conclusion. 
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Table 12 also shows the response rates computed after the first refusal conversion. (If a Screener 
was completed in the second refusal conversion attempt, it is classified as a refusal for this table in order 
to assess the results of the initial refusal treatment.) This table provides support for the combined effects 
of the advance letter mailing treatments and the first refusal conversion treatments, that is, without the 
effect of the second refusal conversion treatment. An examination of the response rates by group given in 
table 12 reveals many of the same patterns discussed earlier in conjunction with table 6. The two groups 
that did not include any payment generally have the lowest response rates. 4 The response rates for groups 
3, 7, 8, 9, and 10 arc all close to 65 percent. Of these five groups, only group 7 involved sending the 
household less than $4 and group 7 had a Priority Mailing at refusal conversion. These findings are 
consistent with previous research in that they show these monetary incentives improve response rates and 
that higher levels of payment ($4 and $5 arc still very modest payments) are typically more effective than 
$ 2 . 



Priority Mail 

Refusing households were sent a refusal conversion letter using Priority Mail after they initially 
refused the screener in several of the groups with the hope that Priority Mail would distinguish the letter 
from other types of mail and convey the importance attached to responding to the survey. If a household 
refused twice and was in a group that had no previous Priority Mailing (groups 1, 2, 3, 6, 9, and 10), then 
75 percent of the households in the group were randomly sampled and these households were sent a third 
mailing using Priority Mail. The remaining 25 percent (like the households in the other groups) were not 
sent any mailing. A discussion of the effectiveness of the initial refusal Priority Mailing is provided 
below, followed by an analysis of the second refusal conversion Priority Mailing in NHES:2003. 

The response rates in table 12 can be used to examine the effectiveness of Priority Mail in the first 
refusal conversion attempt. For example, groups 2 and 5 have the same advance mailing treatment ($0 in 
a first class mail) and the same monetary incentive ($2) for the refusal conversion attempt. They differ 
only in that the refusal conversion letter for group 2 was sent by first class while for group 5 it was sent 
by Priority Mail. The four valid comparisons are: group 1 versus group 4, group 2 versus group 5, group 6 
versus group 7, and group 8 versus group 10. No differences in unit response rates for these contrasts 
were detected. Thus, the experiment shows that using Priority Mail in the initial refusal conversion letter 
does not improve response rates in NHES, holding the other conditions constant. 

Priority Mail was used for the second refusal conversion in order to examine whether its use 
improved the conversion rate. At the second refusal conversion stage, letters were sent by Priority Mail to 
a 75 percent random sample of cases in the groups that had not previously been sent Priority Mail. Those 
that had received a letter by Priority Mail during initial refusal conversion were not sent another letter at 
this stage. Table 13 shows the percentage of the second refusal conversions that were completed for each 
of the six groups by whether the case was in the 75 percent sample sent a Priority Mail letter. Among the 
six groups, the maximum difference in the completion percentage for households sent the Priority Mail 
letter and those not sent any letter was 15 percentage points. When the treated groups are aggregated into 
one category, the difference in the completion percentage between those sent a letter by Priority Mail and 
those not sent any mail in the second refusal conversion is statistically significant. 



4 There was no detectable difference in the rate for group 4 compared to groups 2 and 5 after adjusting for multiple comparisons. 
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